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ABSTRACT 

We  describe  an  efficient  method  for  tracking  humans  in  a  multi-camera  network. 
Based  on  online  clustering,  the  proposed  method  groups  single-camera  tracks  into 
mutually  exclusive  subsets  corresponding  to  individuals.  Experiments  indicate  that 
the  algorithm  is  capable  of  achieving  strong  tracking  accuracy  on  simulated  data. 
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Multi-camera  Human  Tracking  via  Clustering 

Executive  Summary 

We  describe  a  method  for  tracking  humans  in  a  multi-camera  network.  Based  on  online  clus¬ 
tering,  the  proposed  method  groups  single-camera  tracks  into  mutually  exclusive  subsets  corre¬ 
sponding  to  individuals.  By  considering  all  pairwise  similarities  between  single-camera  tracks, 
the  method  aims  to  maximise  within-cluster  similarity,  while  minimising  between-cluster  similar¬ 
ity.  The  method  can  work  in  batch  mode  as  well  as  online  mode  where  it  can  reassign  elements  as 
new  data  arrives. 

Using  a  synthetic  map  and  agents,  and  assumed  distributions  of  similarity  scores,  the  behaviour 
of  the  algorithm  is  examined  under  various  conditions  such  as:  number  of  targets,  number  of 
cameras,  percentage  of  camera  overlap,  probability  of  track  breakage.  In  order  to  assess  the  quality 
of  the  tracking,  we  use  a  novel  error  measure  based  on  the  mutual  information  between  the  ground 
truth  and  the  computed  tracking  result. 

Experiments  show  that  given  a  reasonable  quality  of  single-camera  tracks,  the  algorithm  is 
capable  of  achieving  strong  tracking  accuracy  on  a  number  of  simulated  scenarios.  In  particular, 
it  was  found  that  tracking  accuracy  improves  as:  the  number  of  non-overlapping  cameras  and 
targets  decreases,  more  tracks  are  received,  and  cameras  have  higher  percentage  of  overlap.  These 
findings  indicate  that  the  proposed  method  may  be  suitable  for  real-world  surveillance  systems 
that  are  of  interest.  The  development  of  such  systems  is  the  focus  of  our  future  work. 
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1  Introduction 


The  task  at  hand  is  to  track  humans  across  multiple  cameras.  We  assume  the  most  general  case, 
where  no  knowledge  of  the  camera  network  topology  exists,  the  cameras  arc  not  calibrated  and 
may  overlap  or  may  not  overlap  in  their  fields  of  view.  Note  that  it  is  generally  assumed  and 
required  in  the  literature  that  camera  views  either  overlap  or  do  not  overlap,  there  is  very  little 
existing  work  on  the  heterogeneous  case.  Each  camera  detects  and  tracks  human  targets  producing 
a  set  of  single-camera  tracks.  These  single-camera  tracks  must  be  associated  with  each  other  to 
form  multi-camera  tracks  that  cover  all  cameras. 

The  algorithm  must  be  tolerant  of  single-camera  tracking  errors  since  this  is  by  no  means  a 
solved  problem.  Also  the  algorithm  must  be  an  online  method  that  can  be  updated  incrementally 
as  data  arrives  in  real  time.  Most  of  the  methods  described  in  the  literature  arc  batch  methods 
meaning  that  all  data  from  past,  present  and  future  arc  processed  at  once  to  give  a  globally  optimal 
solution.  This  is  not  practical  for  systems  where  the  user  wants  to  know  the  whereabouts  of  an 
individual  in  the  camera  network  in  real  time. 

Here  we  propose  an  online  clustering  method  that  groups  single-camera  tracks  into  mutually 
exclusive  multi-camera  tracks,  i.e.,  one  for  each  individual.  By  considering  all  pairwise  similar¬ 
ities  between  single-camera  tracks,  the  method  aims  to  maximise  within-cluster  similarity,  while 
minimising  between-cluster  similarity. 


2  Problem  Definition 


Let  T  =  ty  |  he  an  ordered  set  of  N  observed  single-camera  tracks  from  all  cameras.  A 

single-camera  track  (or  simply  track )  t  is  a  description  of  the  path  that  a  single  target  (human) 
tar(t)  takes  in  a  single  camera  cam(t)  field  of  view.  We  assume  that  cam(t)  is  known,  while  tar (f) 
is  not.  Each  track  t  contains  information  such  as: 


•  Track  start  time  start(f). 

•  Track  end  time  end(t)  >  start! r) . 

•  Set  of  target  detections  det(t).  Each  detection  cl  e  det(?)  is  also  associated  with  a  target  tar(d) 
and  camera  cam(d)  (equal  to  cam(t)). 


Our  task  is  to  partition  T  into  M  mutually  exclusive  subsets  C  =  {C i, . . . ,  Cm)  such  that 

(J.C;  -  T  and  C,  n  Cj  =  0  Vi  +  j. 

Cluster  C\  e  C  contains  all  the  tracks  corresponding  to  a  single  target  i  and  together  they  form  a 
multi-camera  track.  Hence  C,  can  be  used  to  track  target  i  in  all  the  cameras.  Note  that  each  track 
t  €  T  belongs  to  exactly  one  cluster  in  C,  which  we  will  call  C{t). 
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3  Clustering 

We  now  describe  a  clustering  method  that  aims  to  find  the  partitioning  C.  The  method  is  online 
meaning  that  it  can  work  as  new  tracks  arrive.  Given  T,  our  task  is  to  find  the  optimal  clustering 
C*\ 


C*  =  argmax5(C),  (1) 

c 

where  5  (C)  is  some  scoring  function  for  clustering  C.  What  form  should  5  have?  We  know  that 
tracks  that  arc  in  the  same  cluster  should  correspond  to  the  same  target  and  hence  must  be  similar. 
Tracks  that  arc  in  different  clusters,  on  the  other  hand,  must  be  dissimilar.  Suppose  we  have  the 
similarity  function  for  track  pairs  S'  :  T  xT  — >  [0, 1],  such  that  4)  is  1  when  t,  and  4  are 
most  similar.  The  score  of  a  clustering  C  can  now  be  written  as: 

S(C)=  ^  S‘(ti,  4)  +  (1  -S\ti,tk))  Vti*tkeT.  (2) 

C(/,)=C(4)  C(ti)*C(tk) 

Note  that  5(C)  is  defined  to  be  0  if  |C|  <=  1.  5(C)  requires  0(N2)  computations,  which  can  be 
costly  when  the  number  of  tracks  N  is  large.  We  now  describe  an  efficient  method  of  computing 
5  (C)  incrementally  as  C  is  slowly  altered. 

Algorithm  1  assigns  a  newly  arrived  track  t  to  a  new  cluster  in  C.  It  then  allows  each  track 
to  move  between  clusters.  In  particular,  we  consider  moving  each  track  p  from  its  current  cluster 
C{p)  to  some  new  cluster  C,.  If  a  move  increases  the  score  5(C)  then  we  keep  it,  otherwise  we 
reject  it.  We  terminate  when  no  single  move  increases  the  score.  Other  types  of  moves  can  also  be 
considered,  however,  experiments  have  shown  that  this  move  is  sufficient  for  finding  a  high-scoring 
clustering. 


Algorithm  1  :  Clustering  algorithm 

Input: 

Current  clustering  C,  newly  arrived  track  t 

1. 

C*  :=  0 

2. 

Do  : 

3. 

improved  :=  False 

4. 

Wp  e  T  : 

5. 

VC,-  £  (C  U  {?})  : 

6. 

Move  p  to  C,  :  C{p)  :=  C(p)\p,  C, 

:=  Ci  U  p 

7. 

If  5(C)  >  5  (C*)  : 

8. 

C*  :=  C 

9. 

improved  :=  True 

10. 

Undo  move:  C{p)  :=  C( p)  U  p,  Cj  : 

=  CAp 

11. 

While  improved 

Output: 

Optimal  clustering  C* 

It  is  important  to  note  that  we  can  avoid  computing  5  (C)  from  scratch  in  step  7  of  Algorithm 
1.  This  is  because  when  we  move  p  from  C( p)  to  some  C,  only  a  small  number  of  terms  in  (2) 
change.  For  example,  consider  Figure  1.  As  p  is  moved,  only  the  contribution  from  the  blue  and 
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red  edges  changes  in  the  computation  of  S  (C):  the  contribution  from  all  other  edges  (dashed)  is 
unchanged.  In  particular,  the  contribution  from  inner-cluster  edges  (blue)  changes  from  +S\p,  •) 
to  1  -  S\p,  •),  while  the  contribution  from  inter-cluster  edges  (red)  changes  from  1  -  S\p,  •)  to 
+S\p,-). 


Figure  1:  Track  p  is  moved  from  its  cluster  C(p)  to  a  new  cluster  Cj.  Edges  between  p  and  tracks 
in  the  same  cluster  are  in  blue,  while  edges  between  p  and  tracks  in  Cj  are  in  red.  Dashed  edges 
are  all  the  remaining  edges  (only  some  are  shown  for  clarity). 


We  will  now  provide  an  empirical  estimate  for  the  running  time  complexity  of  Algorithm  1. 
During  each  iteration  (steps  4  and  5)  we  tty  to  move  every  track  to  every  cluster,  hence  there  are 
NM  such  moves.  In  the  worst  case  each  cluster  contains  N  tracks,  thus  for  each  such  move  we 
need  to  update  the  contribution  of  2 N  -  1  edges.  Empirically  the  total  number  of  iterations  (step  2) 
seems  to  grow  as  O(M).  This  gives  an  overall  running  time  complexity  of  0(M2N2).  Note  that  the 
running  time  can  be  reduced  significantly  by  limiting  the  set  of  tracks  that  can  be  moved  during 
each  iteration.  For  example,  we  can  avoid  moving  old  tracks. 


3.1  Related  Work 


In  traditional  online  multi-camera  tracking  there  is  a  similarity  score  S'(t,  Cj),  which  measures 
the  similarity  between  the  newly  added  track  t  and  an  existing  multi-camera  track  Cj.  Usually 
S  \t,  Cj)  is  defined  as  S  ft,  p),  where  p  is  the  most  recent  track  in  Cj.  The  new  track  t  is  then  added 
to  the  most  similar  multi-camera  track,  as  shown  in  Algorithm  2.  Once  t  has  been  added  to  a 
multi-camera  track  it  remains  there  forever.  This  means  that  “mistakes”  made  early  in  the  process 
(such  as  assigning  t  to  the  wrong  C,)  cannot  be  undone  later  and  undermine  the  quality  of  future 
assignments. 

In  contrast  to  the  traditional  method,  our  method  (Algorithm  1)  optimises  the  (dis)similarity 
between  clusters  in  C.  Furthermore,  our  method  is  able  to  modify  past  “mistakes”,  allowing  it  to 
have  the  most  accurate  assignment  at  any  given  moment  in  time. 
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Algorithm  2  :  Traditional  online  multi-camera  tracking  algorithm 

Input: 

Current  multi-camera  tracks  C,  newly  arrived  track  t 

1. 

bestScore  :=  -oo 

2. 

b:=  0 

3. 

VC,  e  (C  U  {))  : 

4. 

If  S'(t,  Ci)  >  bestScore  Then 

5. 

bestScore  :=  S'(t,Cj ) 

6. 

b  :=  i 

7. 

Ck  :=  Cb  U  t 

Output: 

Optimal  multi-camera  tracks  C 

4  Experiments 


We  used  synthetic  data  to  test  the  clustering  algorithms.  We  simulated  agents  moving  in  a  2D 
world  that  is  observed  by  cameras  with  rectangular  fields  of  view  (Figure  2).  The  camera  views 
may  or  may  not  overlap.  During  each  time  step,  each  agent  produces  a  new  detection  by  moving  to 
a  neighbouring  unoccupied  location  chosen  at  random.  An  agent  may  exit  the  map  if  it  is  located 
near  the  map's  edge  {e.g.,  pink  agent  in  Figure  2).  Each  time  step  there  is  a  probability  of  50% 
that  a  new  agent  will  appear  somewhere  on  the  map.1  A  new  track  is  started  whenever  an  agent 
enters  a  new  camera.  An  active  track  is  terminated  whenever  an  agent  exits  the  camera  of  interest 
(or  exits  the  map).  We  use  a  small  probability  cr  =  10%  that  an  active  track  will  be  terminated  due 
to  malfunctions  in  the  single-camera  tracking  algorithm.  When  this  happens,  a  new  track  is  started 
where  the  previous  one  got  broken.  Old  tracks  have  little  correspondence  with  recent  tracks,  even 
if  they  are  from  the  same  target.  For  example,  imagine  Lying  to  match  a  2-year  old  track  with  a 
1-hour  old  track  of  the  same  person.  For  this  purpose,  our  simulation  removes  all  tracks  that  arc 
older  than  W  events.  Removed  tracks  do  not  affect  the  clustering  score  nor  the  performance  error 
(see  Equation  5).  In  our  simulations  we  set  W  to  25. 

Since  a  track  can  grow  arbitrarily  large,  it  is  not  possible  to  store  every  detection.  Furthermore, 
old  detections  in  a  long  track  have  little  (if  any)  correspondence  with  recent  detections.  To  reflect 
this  we  model  a  track  as  a  first-in-first-out  (FIFO)  list  of  detections  of  maximum  size  Z.  This 
means  that  we  only  store  Z  of  the  most  recent  detections  per  track.  In  our  simulations  we  set 
Z  to  25.  We  compute  the  similarity  of  a  pair  of  tracks  Sl(tj,  tk)  based  on  the  similarity  of  their 
corresponding  detections: 


S‘(ti,tk) 


1 

|det(?,)||det(h;)l 


^  S\di,dk), 

rf,edet(/,).  <iedet(4) 


(3) 


where  Sd{dj,  dk)  is  the  similarity  between  detections  d,  and  dk  in  the  range  [0, 1].  In  the  design  of 
Sd  we  took  into  account  three  factors:  whether  dj  and  dk  belong  to  the  same  target,  whether  they 
occur  in  the  same  camera,  and  also  their  time  difference  5.  Intuitively,  Sd(d,,  dk)  should  be  higher 
when  tar(c//)  =  tar(c4)  and  lower  otherwise.  In  a  real-world  application  we  expect  the  accuracy 
of  Sd(di,dk)  to  be  greater  when  camff/,)  =  camff4)  and  lower  otherwise.  Also  the  accuracy  of 
Sd{dj,  dk)  should  decrease  as  the  time  difference  6  between  the  two  events  increases,  since  the  two 

^his  is  provided  that  the  maximum  number  of  agents  has  not  been  reached  already. 
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File 


Figure  2:  Simulator  example.  This  2D  world  contains  3  agents  (pink,  green  and  blue)  and  2 
overlapping  cameras  ( red  and  green  rectangles).  Active  tracks  are  shown  as  lines  coloured  as 
their  corresponding  camera.  For  example,  green  and  blue  agents  are  currently  being  tracked  in 
both  cameras,  while  the  pink  agent  is  only  tracked  in  a  single  camera. 


cameras  cam (di)  and  cam (74)  are  more  likely  to  be  experiencing  different  lighting  conditions.2 
With  this  in  mind,  we  use  four  similarity  distributions  as  shown  in  Figure  3(a).  These  distributions 
were  chosen  because  they  can  be  easily  sampled.  If  u  is  uniformly  chosen  from  [0, 1]  then  the 
similarity  measure  is  given  as: 


Sd(dj,dk)  =  < 


Ml/(3-0.251og(<5+l)) 

Ml/(2-0.251og(<5+l)) 

1  _  M 1  /(3-0.25  log((5+ 1)) 
1  _  Ml/(2-0.251og(<S+l)) 


if  tar(d;)  =  tar(74)  and  cam (7/,)  =  cam (dk) 
if  tar(d;)  =  tar(74)  and  cam  (di)  7  cam  (dk) 
if  tar(<4)  7  tar(<4.)  and  cam(c/,)  =  cam(7//{) 
if  tar(<4)  7  tar(<4)  and  cam(c/,)  7  cam(74) 


(4) 


In  the  above,  6  affects  the  width  of  the  distributions  as  shown  in  Figure  3(b). 

In  the  experiments  we  sample  Sd(-,  •)  from  one  of  the  four  relevant  distributions.  Once  a  value 
has  been  chosen  it  is  fixed  and  does  not  change  over  time.  In  contrast,  the  value  of  S'it,,  4)  may 
change  over  time  as  new  detections  are  added  to  l,  or  4.  As  a  result,  the  value  of  S\tj,  4)  only 
converges  to  its  true  value  over  time. 

Note  if  tj  and  4  overlap  in  time3  and  they  are  from  the  same  camera,  then  we  know  that  they 
cannot  belong  to  the  same  multi-camera  track  and  hence  we  set  Sr(4  4)  :=  -00.  Every  experiment 
is  evaluated  on  100  trials.  If  C  is  the  true  clustering  for  the  set  of  tracks  T,  then  the  error  for  the 
computed  clustering  C  is: 

2For  example,  one  camera  is  filming  during  the  day,  while  the  other  camera  is  filming  during  the  evening. 

3This  occurs  if  start(f,)  <  end(4)  and  start(4)  <  end (f,). 
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Figure  3:  (a)  The  four  distributions  that  we  sampled  to  obtain  the  similarity  score  Sd{di,dk), 
assuming  the  two  detections  occurred  at  the  same  time,  i.e.,  <5  =  0.  (b)  The  effect  on  one  of  the 
distributions  ( tar(dj)  =  tarfdf)  and  cam{dj)  +  camidf))  as  the  time  difference  5  is  varied  from  0  to 
500. 


E{C,  C)  := 


H(C,C)  :=- 


2 H(C,  C )  -  H(C )  -  H{C) 


where 

\Q\ 


H(C,  C ) 

^log2^and 

C,6C 

IQ  n  ck | 


I 

C/gC,  CgeC 


N 


log2 


N 

IQ  n  Q| 


N 


(5) 


H(C )  is  the  entropy  of  C,  while  H(C,  C)  is  the  joint  entropy  of  C  and  C.  The  numerator  of 
E(C,  C)  is  known  as  variation  of  information  or  shared  information  distance.  E  is  a  metric 4,  with 
E(X,  X)  =  0  and  E(X,  Y )  <  1  VX,  Y. 

We  investigated  how  various  factors  affect  the  performance  of  the  clustering  method  in  Algo¬ 
rithm  1 : 


•  Varying  the  number  of  agents  and  4  cameras:  Figures  4(a-b). 

•  Varying  the  number  of  non-overlapping  cameras  and  5  agents:  Figures  4(c-d). 

•  Varying  the  percentage  of  camera  overlap  for  2  cameras  and  5  agents:  Figures  4(e-f). 

•  Varying  the  probability  of  track  breakage  cr  for  2  cameras  and  5  agents:  Figures  5(a-b). 

•  Varying  the  maximum  track  size  Z  for  4  cameras  and  5  agents:  Figures  5(c-d). 

•  Varying  the  maximum  age  of  a  track  W  for  4  cameras  and  5  agents:  Figures  5(e-f). 

4A  metric  is  a  binary  function  that  satisfies  symmetry,  positivity  and  the  triangle  inequality. 
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For  all  experiments  cr  =  10%,  IV'  =  25  and  Z  =  25  unless  stated  otherwise.  In  general 
the  errors  decrease  over  time.  This  can  be  explained  by  the  fact  that  clusters  become  larger, 
the  (dis)similarity  between  clusters  is  estimated  more  accurately  and  the  elfect  of  noise  in  the 
similarity  scores  is  reduced.  Under  various  conditions  the  clustering  method  is  able  to  recover 
and  maintain  good  clustering.  The  clustering  performance  improves  as  the  percentage  of  camera 
overlap  increases  (Figures  4(e-f)).  Higher  percentage  of  camera  overlap  results  in  a  higher  rate 
of  detections  and  hence  provides  more  information  to  the  clustering  algorithm.  Results  in  Figure 
4  arc  as  expected:  the  error  increases  as  the  number  of  agents  increases,  the  number  of  non¬ 
overlapping  cameras  increases,  or  the  percentage  of  camera  overlap  decreases.  It  may  seem  that 
a  larger  number  of  non-overlapping  cameras  should  decrease  the  error  since  each  target  is  seen 
more  times.  However  this  is  not  the  case,  because  a  larger  number  of  non-overlapping  cameras 
will  produce  a  larger  number  of  tracks  N  -  increasing  the  complexity  of  the  problem  we  arc  trying 
to  solve.  Figures  5(a-b)  show  that  an  increased  percentage  of  track  breakage  does  not  alfect  the 
tracking  accuracy  -  this  seems  counterintuitive  and  at  this  stage  we  cannot  explain  it.  Figures  5(c- 
d)  show  that  it  is  sufficient  to  have  a  rather  small  maximum  track  size  of  5  detections.  Figures 
5(e-f)  show  that  the  error  decreases  significantly  as  W  increases.  From  our  experiments,  the  use  of 
the  constraint  S\ti,  4)  =  -oo  for  simultaneous  tracks  gives  only  a  marginal  improvement  in  error. 


5  Future  work 

If  we  know  that  there  are  exactly  M  targets  then  the  clustering  task  can  be  formulated  via  an  M- 
labeled  graphical  model,  which  can  be  solved  efficiently.  Let  G(T,  £)  be  a  fully  connected  graph 
of  tracks  (/.<?.,  £  =  T  x  T)  and  each  track  t;  takes  the  label  y;  6  {1, . . . ,  M },  then  we  minimise  the 
following  real-valued  energy  function: 

E(y):=  2  Wy^yjWSHkJj),  (6) 

(<J)€6 

where  |[ a  ||  is  an  indicator  function  that  is  0  if  x  =  True,  and  1  otherwise.  Minimising  (6)  is 
now  equivalent  to  maximising  (2).  Importantly,  the  energy  function  (6)  can  be  minimised  using 
multiway  cuts.  This  will  give  us  y*  =  argmirq  E(y)  and  from  this  we  can  obtain  the  clustering  C* 
via  a  simple  Depth  First  Search.  Note  that  for  M  =  2  this  approach  becomes  the  standard  graph 
cuts  algorithm,  which  is  exact  and  has  0(N 3)  running  time  complexity  [Kolmogorov  &  Zabih 
2004].  Our  future  work  will  focus  on  using  the  proposed  method  with  real-world  surveillance 
systems  that  are  of  interest 


6  Conclusion 

We  presented  a  method  for  tracking  humans  in  a  multi-camera  network.  Our  method  is  based 
on  online  clustering,  where  single-camera  tracks  arc  grouped  into  mutually  exclusive  subsets  cor¬ 
responding  to  individuals.  Clustering  is  achieved  by  maximising  a  cost  function  via  a  series  of 
swap  moves.  Importantly,  the  cost  function  takes  into  account  all  the  pairwise  similarities  between 
single-camera  tracks,  thus  utilising  all  the  available  information.  The  method  can  work  in  batch 
mode,  as  well  as,  online  mode  where  it  can  reassign  elements  as  new  data  arrives. 
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Figure  4:  Error  obtained  by  the  clustering  method  for  various  parameter  settings.  Left  figures 
show  the  change  of  error  through  time.  Right  figures  show  the  mean  and  variance  of  error  after  50 
events,  (a-b)  Varying  number  of  agents  and  4  cameras,  (c-d)  Varying  number  of  non-overlapping 
cameras  and  5  agents,  (e-f)  Varying  percentage  of  camera  overlap  for  2  cameras  and  5  agents. 
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Figure  5:  Error  obtained  by  the  clustering  method  for  various  parameter  settings.  Left  figures 
show  the  change  of  error  through  time.  Right  figures  show  the  mean  and  variance  of  error  after 
50  events,  (a-b)  Varying  the  probability  of  track  breakage  cr  for  2  cameras  and  5  agents,  (c-d) 
Varying  the  maximum  track  size  Z  for  4  cameras  and  5  agents,  (e-f)  Varying  the  maximum  age  of 
a  track  W  for  4  cameras  and  5  agents. 
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We  used  synthetic  camera  topology  maps  to  examine  the  behaviour  of  the  algorithm  under 
various  simulated  conditions.  By  varying  each  input  parameter  we  were  able  to  characterise  the 
performace  of  the  algorithm.  Given  a  reasonable  quality  of  single-camera  tracks,  the  algorithm 
was  able  to  achieve  strong  tracking  accuracy  on  a  number  of  simulated  scenarios.  In  particular,  it 
was  found  that  tracking  accuracy  improves  as:  the  number  of  non-overlapping  cameras  and  targets 
decreases,  more  tracks  arc  received,  and  cameras  have  higher  percentage  of  overlap. 
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